Human and Virtual Agents Interacting in the Virtuality Continuum

1 Introduction In this paper we take a multi-party interaction point of view on our research on multimodal interaction between agents in various virtual environments: an educational environment, a meeting environment, and a storytelling environment. These environments are quite different. All these environments require the modeling of multimodal interaction: interactions between human users, the environments and objects represented in the environments, and embodied conversational agents that represent human users or that have been designed to play particular roles in the environment. E.g., such agents play the role of an information or navigation agent, play the role of a meeting assistant in a virtual meeting environment, or play the role of an actor in a virtual storytelling environment. Rather than interacting with one particular user they need to interact with different human and synthetic agents, they need to know about the properties (personalities, intelligence, emotions, capabilities, etc.) of these different agents, they need to know who is aware of what they are saying or doing, and they need to maintain a model of the multi-party dialogue between the different agents. Presently, no one is able to provide such a model of multi-party interaction. On the other hand, more and more we see research projects touching on these topics and for that reason we think it is useful to make our attempts to enter this area of research explicit. In this paper we introduce several projects we work on. The main aim to introduce these projects in one paper is that it makes it possible to compare research approaches on interaction modeling in virtual, mixed-reality and real (physical) environments. For these environments, interaction modeling means multimodal (verbal and nonverbal) interaction modeling and it means (1) modeling human behavior and human – human interaction behavior, (2) modeling human and virtual actor behavior and human – virtual actor behavior, and (3) modeling behavior of virtual actors and virtual actor – virtual actor behavior. In addition, see also [15], it is possible to talk about different types of audience involvement, that is, virtual and real people that mainly observe but may be given the opportunity to influence the activities by their presence and their reactions. When talking about virtual actors we can as well talk about smart objects, smart graphics, and interactions with a smart or ambient intelligence environment. In fact, we are looking at the so-called virtuality continuum as mentioned in [13], but rather …